Papers with Knowledge-based Visual Question Answering
EchoSight: Advancing Visual-Language Models with Wiki Knowledge (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge-based visual question answering systems struggle with these tasks due to limited integration of external knowledge. |
| Approach: | They propose a framework that enables large language models to answer visual questions requiring encyclopedic knowledge. |
| Outcome: | The proposed framework improves retrieval outcomes and accuracy of knowledge-based visual question answering tasks. |
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA). |
| Approach: | They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks. |
| Outcome: | The proposed framework is used to train and evaluate multi-modal retrievers. |
HyperMR: Hyperbolic Hypergraph Multi-hop Reasoning for Knowledge-based Visual Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on knowledge-based visual question answering (KBVQA) describe the semantic distance using the actual Euclidean distance between two nodes, which leads to distortion in modeling knowledge graphs with hierarchical and scale-free structure. |
| Approach: | They propose to use the actual Euclidean distance between two nodes to solve a problem of hierarchical and free-scale knowledge graphs. |
| Outcome: | Extensive experiments on the KVQA, PQ and PQL datasets demonstrate the effectiveness of HyperMR for strong-hierarchy knowledge graphs. |